An Intelligent Network Intrusion Detection System Using Deep Reinforcement Learning

Abstract

Cyberattacks continue to threaten computer networks and information systems, making effective intrusion detection more important than ever. While traditional machine learning and deep learning-based Intrusion Detection Systems (IDS) have achieved encouraging results, they often face challenges in accurately identifying multiple types of attacks with low false alarms. To address this limitation, this study presents a Deep Reinforcement Learning (DRL)-based IDS that uses the Deep Q-Network (DQN) algorithm for multiclass network attack detection. The proposed system classifies network traffic into five categories: Normal, Denial of Service (DoS), Probe, Remote to Local (R2L), and User to Root (U2R). The NSL-KDD dataset was used for training and evaluation, and class imbalance was addressed using the Synthetic Minority Oversampling Technique (SMOTE). The performance of the DQN model was compared against three distinct machine learning paradigms: a supervised Multilayer Perceptron (MLP), an unsupervised Autoencoder, and an ensemble Voting Classifier. The experimental results show that the DQN-based IDS outperformed the other models, achieving 97.96% accuracy, 98.70% precision, 97.96% recall, and an F1-score of 98.24%, along with a low false positive rate of 0.33%. These findings demonstrate that deep reinforcement learning can significantly enhance multiclass intrusion detection while reducing false alarms, making it a promising approach for strengthening network security.

Introduction

This paper presents a Deep Q-Network (DQN)-based Intrusion Detection System (IDS) for multiclass network attack classification. As computer networks and internet services continue to expand, organizations face increasing cybersecurity threats such as data breaches, unauthorized access, financial losses, and service disruptions. While traditional machine learning and deep learning techniques have improved intrusion detection, many existing systems rely on static learning models that struggle to adapt to evolving cyberattack patterns. To overcome this limitation, the proposed system applies Deep Reinforcement Learning (DRL), enabling adaptive learning and decision-making in dynamic network environments.

The proposed IDS uses the NSL-KDD dataset to classify network traffic into five categories: Normal, Denial of Service (DoS), Probe, Remote-to-Local (R2L), and User-to-Root (U2R) attacks. The study compares the DQN model with three baseline approaches representing different machine learning paradigms: a Multilayer Perceptron (MLP) for supervised learning, an Autoencoder for unsupervised learning, and a Voting Classifier for ensemble learning. The research aims to demonstrate improved detection accuracy while reducing false positive rates.

The literature review discusses recent advances in intrusion detection using machine learning, deep learning, hybrid models, and ensemble techniques. Although these methods have achieved high classification accuracy, they depend heavily on labeled training data and fixed learning models. Recent studies have explored reinforcement learning and DRL approaches, including Deep Q-Learning and Dueling Double DQN, showing their potential for adaptive intrusion detection. However, achieving accurate multiclass classification with low false positives and comparing DRL with multiple learning paradigms remain open research challenges.

The proposed methodology consists of data preprocessing, model training, testing, and performance evaluation. During preprocessing, attack records in the NSL-KDD dataset are grouped into the four major attack categories along with normal traffic. Categorical features and labels are converted into numerical values using label encoding, irrelevant attributes are removed, and Min-Max normalization is applied to scale the data. To address the dataset's class imbalance, particularly for R2L and U2R attacks, the Synthetic Minority Oversampling Technique (SMOTE) generates additional samples for minority classes. The dataset is then divided into 80% training and 20% testing sets.

The proposed DQN-based IDS models each network connection as a state represented by 41 input features. The DQN agent selects one of five possible attack classifications as its action and receives rewards for correct predictions and penalties for incorrect ones. An experience replay buffer stores interactions (state, action, reward, and next state), allowing the neural network to continuously improve its decision-making policy through reinforcement learning. This adaptive learning process enables the IDS to respond more effectively to evolving attack patterns than traditional supervised models.

To ensure a fair comparison, the MLP, Autoencoder, and Voting Classifier are trained using the same preprocessed dataset and evaluated with identical metrics, including accuracy, precision, recall, F1-score, false positive rate, and confusion matrix analysis.

Conclusion

This study presented a Deep Q-Network (DQN)-based intrusion detection system for multiclass network attack classification. The approach was designed to accurately identify Normal, DoS, Probe, R2L, and U2R traffic while keeping false alarms to a minimum. In the experimental evaluation, the DQN model consistently outperformed the MLP, Autoencoder, and Voting Classifier models across all evaluation metrics. These results suggest that the reward-driven learning mechanism of deep reinforcement learning is well suited to capturing complex network traffic patterns and improving intrusion detection performance. Beyond its strong classification results, the proposed model also maintained a low false positive rate, an important requirement for any practical intrusion detection system. Being able to detect multiple attack categories accurately while reducing false alarms highlights the potential of deep reinforcement learning for strengthening network security and supporting more reliable threat detection. There are several promising directions for future work. The proposed approach could be evaluated on more recent intrusion detection datasets and tested in real-time network environments. More advanced deep reinforcement learning techniques, such as Double DQN and Dueling DQN, could also be explored to further improve detection performance and learning efficiency. Finally, future studies may focus on improving the detection of minority attack categories and integrating the model into live network monitoring and security systems.

References

[1] T. Wisanwanichthan and M. Thammawichai, “A Double-Layered Hybrid Approach for Network Intrusion Detection System Using Combined Naive Bayes and SVM,” IEEE Access, vol. 9, pp. 138432-138450, 2021, doi: 10.1109/ACCESS.2021.3118573. [2] I. Abrar, Z. Ayub, F. Masoodi and A. M. Bamhdi, “A Machine Learning Approach for Intrusion Detection System on NSL-KDD Dataset,” in Proc. Int. Conf. Smart Electronics and Communication (ICOSEC), Trichy, India, 2020, pp. 919-924, doi: 10.1109/ICOSEC49089.2020.9215232. [3] A. Halbouni, T. S. Gunawan, M. H. Habaebi, M. Halbouni, M. Kartiwi and R. Ahmad, “Machine Learning and Deep Learning Approaches for CyberSecurity: A Review,” IEEE Access, vol. 10, pp. 19572-19585, 2022, doi: 10.1109/ACCESS.2022.3151248. [4] G. Muhammad, M. S. Hossain and S. Garg, “Stacked Autoencoder-Based Intrusion Detection System to Combat Financial Fraudulent,” IEEE Internet of Things Journal, vol. 10, no. 3, pp. 2071-2078, 2023, doi: 10.1109/JIOT.2020.3041184. [5] S. K. Gupta, M. Tripathi and J. Grover, “Hybrid Optimization and Deep Learning Based Intrusion Detection System,” Computers and Electrical Engineering, vol. 100, p. 107876, 2022, doi: 10.1016/j.compeleceng.2022.107876. [6] V. Hnamte and J. Hussain, “DCNNBiLSTM: An Efficient Hybrid Deep Learning-Based Intrusion Detection System,” Telematics and Informatics Reports, vol. 10, p. 100053, 2023, doi: 10.1016/j.teler.2023.100053. [7] S. Seth, K. K. Chahal and G. Singh, “A Novel Ensemble Framework for an Intelligent Intrusion Detection System,” IEEE Access, vol. 9, pp. 138451-138467, 2021, doi: 10.1109/ACCESS.2021.3116219. [8] O. Arreche, I. Bibers and M. Abdallah, “A Two-Level Ensemble Learning Framework for Enhancing Network Intrusion Detection Systems,” IEEE Access, vol. 12, pp. 83830-83857, 2024, doi: 10.1109/ACCESS.2024.3407029. [9] V. Sidharth and C. R. Kavitha, “Network Intrusion Detection System Using Stacking and Boosting Ensemble Methods,” in Proc. Third Int. Conf. Inventive Research in Computing Applications (ICIRCA), Coimbatore, India, 2021, pp. 357-363, doi: 10.1109/ICIRCA51532.2021.9545022. [10] M. A. Hossain, “Deep Q-Learning Intrusion Detection System (DQ-IDS): A Novel Reinforcement Learning Approach for Adaptive and Self-Learning Cybersecurity,” ICT Express, vol. 11, no. 5, pp. 875-880, 2025, doi: 10.1016/j.icte.2025.05.007. [11] M. Lopez-Martin, B. Carro and A. Sanchez-Esguevillas, “Application of Deep Reinforcement Learning to Intrusion Detection for Supervised Problems,” Expert Systems with Applications, vol. 141, p. 112963, 2020, doi: 10.1016/j.eswa.2019.112963. [12] H. Alavizadeh, H. Alavizadeh and J. Jang-Jaccard, “Deep Q-Learning Based Reinforcement Learning Approach for Network Intrusion Detection,” Computers, vol. 11, no. 3, p. 41, 2022, doi: 10.3390/computers11030041. [13] H. Gülmez and P. Angin, “A Study on the Efficacy of Deep Reinforcement Learning for Intrusion Detection,” Sakarya University Journal of Computer and Information Sciences, vol. 4, pp. 11-25, 2021, doi: 10.35377/saucis.04.01.834048. [14] Y. Badr, “Enabling Intrusion Detection Systems with Dueling Double Deep Q-Learning,” Digital Transformation and Society, vol. 1, 2022, doi: 10.1108/DTS-05-2022-0016. [15] Y.-F. Hsu and M. Matsuoka, “A Deep Reinforcement Learning Approach for Anomaly Network Intrusion Detection System,” in Proc. IEEE 9th Int. Conf. Cloud Networking (CloudNet), Piscataway, NJ, USA, 2020, pp. 1-6, doi: 10.1109/CloudNet51028.2020.9335796. [16] M. Tavallaee, E. Bagheri, W. Lu and A. Ghorbani, “A Detailed Analysis of the KDD CUP 99 Data Set,” in Proc. Second IEEE Symp. Computational Intelligence for Security and Defense Applications (CISDA), 2009. [17] N. V. Chawla, K. W. Bowyer, L. O. Hall and W. P. Kegelmeyer, “SMOTE: Synthetic Minority Over-sampling Technique,” Journal of Artificial Intelligence Research, vol. 16, pp. 321-357, 2002, doi: 10.1613/jair.953. [18] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, et al., “Human-Level Control Through Deep Reinforcement Learning,” Nature, vol. 518, no. 7540, pp. 529-533, 2015, doi: 10.1038/nature14236.

Copyright

Copyright © 2026 Sana Fathima, O. B. V. Ramanaiah. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.